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Abstract 

We consider a neural network with adapting synapses whose dynamics can be ana- 
litically computed. The model is made of N neurons and each of them is connected 
to K input neurons chosen at random in the network. The synapses are n-states vari- 
ables which evolve in time according to Stochastic Learning rules; a parallel stochastic 
dynamics is assumed for neurons. Since the network maintains the same dynamics 
whether it is engaged in computation or in learning new memories, a very low proba- 
bility of synaptic transitions is assumed. In the limit N —* oo with K large and finite, 
the correlations of neurons and synapses can be neglected and the dynamics can be 
analitically calculated by flow equations for the macroscopic parameters of the system. 



PACS numbers: 87.10.+e, 05.20.-y 
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1. Introduction. 

In the dynamics of Attractor Neural Networks (ANNs) two processes take place: the 
time evolution of the neuron states and the change in strength of the connections (synapses) 
between neurons. Most of the prior research on has focused separately on the first dynamical 
process (retrieval of learned patterns) or the other (learning of patterns) In this work 
we deal with the problem of building up models where neurons and synapses maintain the 
same dynamics whether the network is engaged in computation or in learning new memories; 
the two stages (computation and learning) differ only for the presence of external stimuli. 
This problem is important for the description of the short term memory in the human brain 
(see e.g. [§]) and has been studied mainly in the context of Hebbian learning with a decay 
term Shinomoto || presented a rule for the synaptic modification whose stable 

solutions are the Hopfield couplings || ; Dong and Hopfield f|] considered analog neurons in a 
deterministic network described by a system of differential equations and applied their model 
to the problem of the development of synaptic connections in the visual cortex; D'Autilia 
and Guerra studied an ANN with adapting synapses in order to model conditioned reflex 
and rhythm recognition. 

We consider here an ANN with asymmetric adapting synapses whose dynamics consists 
of a Stochastic Learning mechanism J7|||; in particular we refer to Amit and Fusi || for 
a discussion on the possible sources of stochasticity in the synaptic learning. We present a 
dynamical theory in terms of deterministic flow equations for macroscopic order parameters, 
which holds if the connections are strongly diluted. For quenched synapses the dynamical 
approach is exact in the case of strong dilution 0; it may be mentioned, in passing, that 
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recently Coolen and Sherrington described dynamically the fully connected Hopfield model 

Our model is made of iV neurons (Ising spins) Si(t) 6 { — 1, 1}, i = 1,...,N. For each 
neuron Sj, K input sites ■ ■ ■ , 3k{^) are chosen at random among the N sites, and 

NK synaptic interactions Jij(t) are introduced. We assume that the synapses are n-states 
variables J. t j G {1, rL ^f, • • • j ~~ !}■ Let us call a the index identifying the different values of 
the synapses, so that a = 1 implies J = 1 and a = n implies J = —1; in general 

n+l-2a 

J a = , a = l,...,n. (1) 

n — 1 

The dynamical rule for the evolution of the synapses is assumed as follows. Each synaptic 
variable Jjj connecting Sj with one of its K input neurons grows by one unit -^r, i.e. J a — > 
Ja-i, with probability q if the product SjSj of the spins connected by Jy is positive; if 
such product is negative the synapse decreases by one unit (J a — > J a +i) with the same 
probability. If a synapse is at one of the extreme limits and should be pushed off, its value 
remains unchanged. 

A parallel stochastic dynamics with inverse temperature j3 = T _1 is assumed for neurons, 
where the local field acting on neuron Sj is given by 

Wt)=Y,M t ) s i( t ) + H i(P) ; ( 2 ) 
j 

the sum is taken over the K input neurons and external stimuli are represented by local 
magnetic fields Hi. Therefore the rules for the parallel dynamics in our model are: 

Si(t + 1) = sign (- + ~tanh(/3hi(t)) - rj^j (3) 



Jijit + 1) = J»(t) + —— Si (t) Sj (t)9(q - rjij), (4) 

where r\{ and are random numbers in [0, 1], 9 is the Heaviside function. 

As to the value of q, we assume q = O(j^). This choice deserves a comment. It is 

known |F] || that if q is order or greater, then it takes only one iteration to imprint 

the pattern corresponding to a given neuronic state; since the synapses maintain the same 

dynamics during the retrieval of a previously learned pattern, such occurrence would destroy 

the associative capability of the network; therefore in the sequel we assume that q is order 
j_ 

A' ' 

Let us first describe qualitatively the behavior of our model in absence of external stimuli. 
At low temperature the dynamics drives the system towards one of the 2^ configurations 
satisfying Jij = SiSj (separable configurations). If the initial configurations of neurons and 
synapses are completely disordered, then at late times the network converges to one state 
chosen among the separable configurations; in other words the network spontaneously breaks 
the symmetry among the separable configurations (a similar spontaneous symmetry breaking 
is discussed in [|J). If, on the other hand, the initial conditions are sufficiently polarized 
with respect to a given separable configuration, we expect the network to converge to that 
configuration. 

Our aim is to describe macroscopically the evolution of the system by flow equations 
for the macroscopic parameters describing the state of neurons and synapses. Let us take 
Hi = (no external stimuli on the network) and consider an arbitrary pattern for the 
neurons. We introduce the variables Sj = £jSj and = Jij^j, which will be useful to 
measure the degree of polarization of neurons and synapses with respect to the pattern {^}. 
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The statistical ensemble we deal with consists of all the hystories of the network with the 
initial conditions {sj(0)} {Jij(0)} sampled independently as follows: 

Prob[ 8i (0) = $ = ~(l + mo) (5) 

Prob[J i:j (0) = J a ^j] = Po(«) (6) 

Since the initial conditions are uniform (i.e. independent of the site index i), the statistical 
properties of our model remain uniform at every later time t. We also remark that at t — 
both neurons and synapses are uncorrelated. 

We study the model in the limit N — > oo with K large and finite. It can be shown that 
in such limit the neurons never correlate ||. Indeed the K f sites which belong to the tree 
of ancestors of a given neuron Sj are all different (this is true as long as t << logN). Also 
the correlations between synaptic variables can be neglected: the correlation between two 
synapses Jy and Jik sharing a common site increases when they are simultaneously updated 
while decreases when one out of the two is updated. The probability for the simultaneous 
updating is order q 2 , while the probability for the single updating is order q. Since q is order 
-g, it follows that, at least in the early stage, the synapses can be treated as uncorrelated. In 
the sequel, analyzing in particular the cases n = 2 and n = 3, we will show that actually the 
correlation between synapses can always be neglected. We also remark that Sj is independent 
of Jij because there are no loops in the tree of ancestors of Sj. Therefore neurons (as well as 
synapses) can be treated as independent and identically distributed stochastic variables. 

We now introduce the order parameter m(t) =< Si(t) >= -^J2i< Si(t) > for neurons 
and the probability p(a,t) that Jij(t) is equal to J a . We have, from (H)©, p(a, 0) = po(a) 
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and m(0) = rriQ. The flow equation for p(a,t) is easily found to be 



p(a, t + 1) = T aa> (m(t)) p(a', t), 



(7) 



where the n x n stochastic transition matrix T is tridiagonal with the following structure 



/ 1 - a b 
a 1 — q b 



T(m(t)) 



\ o o 

where a = f (1 - m 2 (t)) and &=§(! + m 2 (t)). 



\ 




a 1 — q b 

a 1 - b J 



In order to calculate the flow equation for the order parameter m(t), we observe that the 
projection on {£} of local field (0) can be written as 



(9) 



where the variable = JijSj is the contribution to the synaptic input from a single input 
neuron. The variable h (the total synaptic input) is the sum of K independent and identically 
distributed stochastic variables J'. The probability distribution for J' reads: 

p'{a, t) = Prob [j;.(t) = J a ] = i(l + m{t))p{a, t) + i(l - m(t))p(n + 1 - a, t), (10) 

where the first term on the RHS of eq.(|II]) is the probability that 5=1 and J — J a , 
the second term is the probability that s = — 1 and J = —J a . The stochastic variable 
hi{t) can assume (n — 1)K + 1 possible discrete values in [— K,K]; its distribution 7it(a) = 
Prob [hi(t) = a] can be calculated by performing K times the convolution of the distribution 
(10|). For example, in the case n = 2, we have 



Ht(a) 



K \ K + a K-, 

V(M)^p'(-M)^ 



K-a 



(11) 



It is now straightforward to evaluate, in the general case, the flow equation for m(t) using 
eqs.(§: 

K 

m(t + 1) = J2 n t( a ) tanh(pa) = < tanh(ph) > Ht (12) 

a=-K 

Flow equations (0) and (|T2|), with initial conditions m and po(ce), describe the coupled 
dynamics of m(t) and p(a,t). 

After this general introduction and the description of the main dynamical equations, we 
will devote the next section to study the fixed points of the dynamics for our model. In 
section 3 we will report the results obtained simulating numerically the flow equations in 
the cases n = 2 and n — 3. In section 4 it is shown that the correlations among synapses 
can always be neglected. In section 5 we study the learning properties of the network. In 
section 6 the simulations of an ANN with adapting synapses are studied and compared to 
the theory. Section 7 summarizes the conclusions. 



2. Stationary solutions. 

Let us now discuss the stationary solutions of the flow equations. First of all we observe 
that the invariant distribution with respect to (0) is given by 

/ \ TlfXi. , O \ Q — J_ / Q \ ft — O/. / \ 

Pm (a = 7— n ^v? 1 -™ (l + m) 13 

(1 + m A ) — (1 — m A ) 

We remark that the stationary distribution does not depend on q. We call TiS m '{a) the 
probability distribution of h corresponding to p m . For stationary solutions the following 
equation holds: 

m =< tanh(ph) > H ( m ) (14) 
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It can be shown that m = is always a stable solution for (|14D , and it is unique for 
small values of (3 (high temperature). As f3 increases, equation (|l4l) displays a first-order 
transition, i.e. for f3 > (3 C (K) two solutions with m > appear discontinuously, the one 
with larger m being locally stable for variations in m. This solution corresponds to a state 
which deviates only slightly from pattern {£}; m = 1 is a stationary and stable solution only 
in the limit (3 — > oo. We have thus shown that the model possesses a stable fixed point in 
correspondence to any pattern {£}. If the initial conditions m and po(ce) are sufficiently 
polarized with respect to {£}, then, by iterating the flow equations, the stable solution 
m s > is asymptotically achieved, i.e.: 



In the case of a large number of connections (large K) a useful approximation can be used. 
Since the local field hi(t) is the sum of K independent and identically distributed stochastic 
variables, we use the central limit theorem to approximate it by a Gaussian random variable 
with mean and variance given respectively by 



lim m(t) = m s 



lim p(a, t) = p ms (a) 

t— >oo 



(15) 



fi(t) = Km{t) < J{t) > 



(16) 



and 




(17) 



The flow equation ( |T2D can be approximated as follows: 




(18) 
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In the zero temperature limit (j3 — > oo) equation fll8|) reads 

m(t+l) = erf Oj&Y (19) 
where ^ may be seen as the signal-to-noise ratio (SNR) for the synaptic input of a neuron. 



3. Analysis of the cases n = 2 and n — 3. 

In this section we study numerically the behavior of the flow equations (0) and ([Tj|) in 
the cases n = 2 and n = 3. The probability distribution for two-state synapses is determined 
by its average J{t) =< J(t) > and flow equations reduce to: 

J(t+1) = (l-q)J(t)+qm 2 (t) (20) 

m(t + 1) = J JL e -^ z2 tanh[/3 (KJ(t)m(t) + (1 - J 2 (t)m 2 (t)) z^j ]. (21) 
The stationarity conditions are 

J = m 2 (22) 

J -^=e-^ z2 tanh f3 (^Km 3 + yj K{1 - m 6 ) z) (23) 



m 



Equation ( p3|) displays a first-order transition for /3 greater than a critical coupling which 
has the behavior f3 c (K) ~ for large K. Hence for (3 > (3 C equations (|20| ) (|2T| ) have two 
stable fixed points, a fixed point with m > and the trivial one. By numerical simulations we 
found that the recurrence equations (^Up fl2~T| ) never show complex behavior and by iterating 



them, starting from any initial condition, it always happens that one of the two stable fixed 
points is approached. 



It is interesting to compare the retrieval capability of our neural network with adapting 
synapses to the case of a strongly diluted network with fixed synapses, where each neuron 



has K random input sites, described by equation (^T|) with J{t) = j7o for every t. The 
process of retrieval in our model will be referred to as adaptative retrieval (AR) whereas the 
retrieval with fixed synapses will be called fixed synapses retrieval (FSR). We remark that 
FSR actually corresponds to the early stage of AR, due to the very low value of q. In fig.l 
we depicted the asymptotic value m(oo) of the order parameter versus the initial condition 
Jq for the two cases and various temperatures. We see that in the case of AR a first order 
transition occurs; in other words, for J7q greater than a threshold value Jth-, the non-trivial 



fixed point of equation ( |23|) is asymptotically achieved; on the other hand a second order 
transition occurs in the case of FSR. For j7o greater than J t h and Jo — Jth small, AR performs 
better than FSR. We observe that a threshold value exists also for m , i.e. for tuq smaller 
than a threshold value m t h (numerically very small), the flow equations (|20|)(|2TD lead to the 
trivial fixed point even for j7o = 1- 

Let us now consider the case n — 3, when synapses take values in {1,0,-1}. Two 
independent parameters are needed in order to specify the distribution of the synapses; let 
us define p + {t) = p(l,t), p~(t) = p(—l,t) and p°(t) = p(0,t). The stationarity conditions 
can be computed in this case with the results: 

(1 + m 2 ) 2 Q 1 — m 4 _ (1 — m 2 ^ 2 



m 



dz 



P + = \ ~ 4 P =V— — P~= \, 4 ( 24 ) 

3 + m 4 3 + m 4 3 + m 4 

:e'^ z2 tanh[/3 (Km(p + - p~) + ^ K(l - p° - m 2 (p+ - p~f) z^j ]. (25) 



/2tt 

The same qualitative description of the n = 2 case applies to the behavior of the flow 
equations in this case. The critical coupling has the behavior f3 c (K) ~ ^ for large K. In 
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fig.2 we depict, for three values of the temperature, the threshold value of Pd = P + — p~ 
versus p°. The threshold p% is such that the non-trivial fixed point is achieved if the initial 
conditions satisfy pn > Pd- 

4. The synaptic correlations. 

In this section we show that, during the time evolution of the network, the correlations 
among the synapses can be neglected. This assumption is tested by taking into account 
the correlation between pairs of synapses sharing a common site. Consider the local field hi 
acting on the neuron Si given by equation (||). The mean and variance of hi are given by: 

p = K < J > m (26) 

a 2 = K (< J 2 > - < J > 2 m 2 ) + K(K - l)m 2 C (27) 

where C =< JijJik > — < J-ij >< Jik > is the pair synaptic correlation. The first term on 
the RHS of eq.(p7|) is the variance of the local field which is computed as if the synapses 
were uncorrelated. 

In the n = 2 case we consider the probability distribution for pairs of synapses {Jij, Jik} 
sharing a common site, which evolves according to a 4 x 4 transition matrix. After a little 
algebra we obtain the flow equations for J (=< >=< J ik >) and C: 

J(t+l) = (l-q)J(t)+qm 2 (t) (28) 

C(t +!) = (!- q) 2 C(t) + q 2 (m 2 (t) - m\t)) (29) 
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while equation fll8|) can be used for the evolution of the order parameter m(t); the initial 



condition for the correlation is C(0) = 0. Equation (|29|) has a clear meaning: the pair 
correlation does not change if the two synapses are not updated (probability (1 — q) 2 ); it 
goes to zero if one out of the two is updated (probability 2g(l — q)); it becomes m 2 — m 4 
if both synapses are updated (probability q 2 ). We studied numerically eqs. (p8|), (p9|) and 
(|I8|) and found that the second term on the RHS of equation fl2"T|) is always very small 
compared to the first term. In fig. 3 we depict the time evolution of the order parameter, for 
a particular set of initial conditions, in two cases: taking into account the pair correlations 
and not. The two curves are indistinguishable; the same behavior is observed for any choice 
of the initial conditions. In fig. 3 we also depict the time evolution of the two contributions 
to the variance of the local field, given by eq . fl27D . The contribution due to the synaptic 
correlations is always very small and reaches a maximum value during the retrieval process. 
The stationarity conditions for eqs. fl28|) and read: 



J = m 2 , C = -^—{m 2 -m 4 ), (30) 
2 — q 



and, at leading order in K, the equation for the fixed points is: 



m 



J ^L e -h* 2 tanh (3 (^Km 3 + ^ K(l - m 6 + ™* ^ ^ ) zj (31) 



to be compared with the equation which is computed as if the synapses were uncorrelated, 
i.e. eq . (|2~3"D . We see that the distance between the non-trivial fixed points of eqs. (|3l|) and 



(p3|) vanishes for large K. Hence the synaptic correlations can be neglected also with respect 
to the equilibrium properties of the network. 

We tested the assumption also in the n = 3 case (see the Appendix). We find that also 
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in this case the contribution to a 2 due to the correlations is always very small and can be 
neglected. In fig. 4 we depicted the time evolution of the order parameter m, in a particular 
history, calculated in two cases, i.e. with and without correlations. Also in this case the two 
curves are indistinguishable. 



5. The learning stage. 

The learning stage is characterized by the presence of external stimuli represented by 
local magnetic fields in equation (g). We recall the results obtained in || about the capacity 



of a stochastic learning network with q of order J and with n arbitrary. Let us suppose 
that the network receives an uninterrupted flow of uncorrelated patterns to be learned: 

... , {0-2, {O-i, {£}o, {£h, {£} 2 , ••• 

In P] it is shown, by a signal-to-noise analysis, that the network acts as a palympsest 



11], [12], i.e. patterns learned far in the past are erased by new patterns, and that the 



maximum number of patterns which can be stored is proportional to \J K |[13 



In our case q is order 4> and each pattern {£} p (p = 0, ±1, ±2, . . .) has to be presented at 



least for I = O [s/K logKJ iterations in order to be stored. Let us consider for example the 
pattern {£}o, an d let us set t = in correspondence of the first presentation of {£}o- Initially 
the synapses have a completely random distribution with respect to {£}o, i- e - p( tt ,0) = -• 
During the presentation of pattern {£}o the synapses are polarized due to the action of 
matrix (|8|) with m — 1, because {s} = {£}()• Subsequently the network begins to learn other 
patterns and the synaptic distribution becomes depolarized by the action of matrix (|8|) with 
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m = 0. After the presentation of z other patterns the synaptic distribution is given by 



where the dependence of T on q has been made explicit. Incidentally we remark that during 
the learning stage the correlation among synapses is identically zero due to the ± symmetry. 
For fixed m and q order h we have 



as can be easily checked performing the Taylor expansion of the matrix T q in a neighborhood 
of q = 0. Therefore rescaling the time by the factor £ and changing q into q = £q leads to 
the same problem studied in ||. Hence we conclude that our network is a palympsest and 



6. Simulations. 

Our theory works as long as t is small with respect to logN, so as to avoid the effects due 
to the neuronic correlations. In order to build up finite-size networks, where the neuronic 
correlations can be neglected for longer times, and to test our dynamical equations, we adopt 
the following approach: we implement a model of N neurons, where each neuron has M input 
neurons chosen at random (M « N). The NM synapses obey the stochastic learning rules 
and, for every neuron and for every time t, K input neurons are chosen at random among 
the M corresponding to Sj and only those K inputs contribute to the local field acting on 
Si at time t. This is a generalization of our model (recovered when K = M) which is ruled 



p(t=(z + 1)£) = Tf(m = 0)7j (m = l)p(0) 



(32) 




(33) 
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by the same dynamical equations. We implemented a N = 10,000 network with M = 200, 
K = 21 and two-states synapses. The time evolution of the macroscopic parameters of the 
system is found to be in agreement with the theoretical estimate until t ~ 20. In fig. 5 
we depict the time evolution of the order parameter and of the synaptic distribution and 
compare it with the theoretical estimate; the initial conditions were m = 1 and Jo = 0.3. 
The agreement with the theory has been tested with many different initial conditions. We 
also implemented a network with N = 10, 000 and M = K — 21 but the time evolution in 
this case was in agreement with the theory only for a few time steps. 

7. Conclusions. 

In this paper we have considered a neural network with clipped synapses and stochastic 
learning rules whose learning capabilities have been studied in [RJ. We extend the analysis 
of this model and investigate the consequences of the synaptic dynamics in the process of 
retrieval. We find that, in order to preserve the associative capability of the system, the 
synaptic transition probability q must be very small; moreover, for strong dilution, the 
dynamics of the network can be analitically calculated because for very small values of q the 
correlations among the synaptic variables can be neglected. As to the learning properties, 
the network acts as a palympsest and the maximum number of storable patterns coincides 
with the result obtained in ||, the only difference being that a pattern has to be presented 
for many iterations in order to be stored. 

In this framework the two stages (computation and learning) differ for the duration of 
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the external stimulus corresponding to a given pattern. If the pattern is presented for a 
sufficiently long time, the network stores it in the synaptic couplings. If the pattern (or a 
damaged version of it) is presented for few iterations, the dynamics of the network is capable 
to reconstruct the pattern provided it had been learned previously (unless new external 
stimuli impinge on the network during the retrieval). 

It would be interesting to study the coupled dynamics of neurons and synapses, assuming 
a stochastic learning mechanism, in a fully connected network. 

Appendix 

We write here the flow equations, in the n = 3 case, which take into account the pair 
synaptic correlations. Let us consider the distribution of a pair of synapses {Jij, Jik] sharing 
a common site, which evolves according to a 9 x 9 transition matrix. We define 

p + = Pr[J l3 = 1] = Pr[J lk = 1] 

p- = Pr[J l3 = -1] = Pr[J tk = -1] 
p° = Pr[Jij = 0] = Pr[J lk = 0] 
p +0 = Pr[Jij = 1, J ik = 0] = Pr\J l3 = 0, J lk = 1] 
p+- = p r \J i3 = 1, J ik = -1] = p r [J tJ = -1, J lk = 1] 
p~° = Pr\J l3 = -1, J ik = 0] = Pr[J l3 = 0, J lk = -1]. 

Next, we have 

< J 2 >= p+ + p_ 
16 



< J >= p+ - p- 

C =< JijJik > - < Jij >< J lk >= p + + p- - p +0 - p-° - Ap + - - (p + - p- f. 

The flow equations for p + , p~ , p° are given by (0), while the flow equation for m(t) is (|T8|). 
The other flow equations are: 

p+°(t+l) = {a 1 -a 3 )p + %t) + {a 2 -a 3 + a 4 -a 5 )p + -{t) + {a±-a 2 )p~ {t) + {a 3 +a 5 )p + {t) + a 2 p°{t) 

(34) 

p + -(t + 1) = a 3 p +0 (t) + (oi + a 2 + a 3 + a 5 )p + -(t) + a 2 p~°(t) + a 5 p°(t) (35) 

p-°(t+l) = {a, ) -a 3 )p + \t) + {a 3 -a 2 )p + ~{t) + {a l ~a 2 )p-\t) + a 3 p {t) + {a 2 + a 5 )p-{t) (36) 

where a\ = (1 — q) 2 , a 2 = \q(l — q)(l + m 2 ), a 3 = |g(l — q)(l — m 2 ), a 4 = \q 2 (l + 3m 2 ) and 
a 5 = \q 2 (l - m 2 ). 
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Figure Captions 



Figure 1: Numerical evaluation of the asymptotic order parameter, versus the initial polar- 
ization Jo of the synapses, with K = 100, q = 0.01, n = 2 and three different values of the 
temperature. The continuous line and the dotted one correspond respectively to AR and 
FSR (see the definitions in the text). 

Figure 2: The threshold value of p D is plotted versus p° in the case n = 3, K = 100, q = 0.01 
and with three different values of the temperature. 

Figure 3: The time evolution of the order parameter evaluated by the flow equations which 
neglect the correlations (•) and by the flow equations which take into account the pair 
synaptic correlation (o) in the case n = 2: the two curves are indistinguishable. The initial 
conditions are m = 0.88, J = 0.55; K = 100, q = 0.01, (3 = 0.03. In the figure we also 
depict the time evolution of the two contributions to the variance of the local field (see the 
text): the contribution due to the synaptic correlations, i.e. (K — l)m 2 C ((}), compared to 
the term which is computed as if the synapses were uncorrelated , i.e. 1 — J 2 m 2 (A). 

Figure 4: The time evolution of the order parameter evaluated by the flow equations which 
neglect the correlations (•) and by the flow equations which take into account the pair 
synaptic correlation (o) in the case n = 3: as in Fig.3 the two curves are indistinguishable. 
The initial conditions are m = 0.8, p D = 0.3, p° = 0.33; K = 100, q = 0.01, (3 = 0.05. In 
the figure we also depict the time evolution of the two contributions to the variance of the 
local field (see the text): the contribution due to the synaptic correlations, i.e. (K — l)m 2 C 
((}), compared to the term computed as if the synapses were uncorrelated , i.e. 1 — p° — 
(p + - p~) 2 m 2 (A). 

Figure 5: Time evolution of the order parameter < s > and the mean of the synapses 
< J > in a ANN with two-states adapting synapses. The numerical results for < s > (•) 
and < J > ((}) are averaged over 80 hystories of a network with iV = 10, 000, M = 200, 
K = 21, q = 0.01 and zero temperature; the initial conditions are mo = 1 and j7o = 0.3. The 
theoretical estimate by the flow equations is represented by o (< s >) and by A (< J >). 
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